On Continuous-Action Q-Learning via Tile Coding Function Approximation

Reinforcement learning (RL) is a powerful machine-learning methodology that has an established theoretical foundation and has proven effective in a variety of small, simulated domains. There has been considerable work on applying RL, a method originally conceived for discrete state-action spaces, to problems with continuous states. The extension of RL to allow continuous actions, on the other hand, has seen relatively little research. One proposed approach to allowing continuous actions is to represent the value function using a tile-coding function approximator. We introduce a simulated domain for the controlled study of this method in conjunction with Q-learning and report empirical results on its performance under different parameterizations. Our experimental findings contribute a deeper understanding of the workings of tile coding in continuous-action domains, provide guidance to parameter choices, and point out an improvement on this method which we verify empirically.